CMS PAPER CFT-09-020 



CMS Paper 



2010/01/11 



Commissioning of the CMS High-Level Trigger with 

Cosmic Rays 

The CMS Collaboration 



Abstract 

The CMS High-Level Trigger (HLT) is responsible for ensuring that data samples with 
potentially interesting events are recorded with high efficiency and good quality. This 
paper gives an overview of the HLT and focuses on its commissioning using cos- 
mic rays. The selection of triggers that were deployed is presented and the online 
grouping of triggered events into streams and primary datasets is discussed. Tools 
for online and offline data quality monitoring for the HLT are described, and the op- 
erational performance of the muon HLT algorithms is reviewed. The average time 
taken for the HLT selection and its dependence on detector and operating conditions 
are presented. The HLT performed reliably and helped provide a large dataset. This 
dataset has proven to be invaluable for understanding the performance of the trigger 
and the CMS experiment as a whole. 



*See Appendix A for the list of collaboration members 
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1 Introduction 

The Compact Muon Solenoid (CMS) detector JT| is one of two large general-purpose experi- 
ments at the CERN Large Hadron Collider (LHC) 0. The 12,500 ton detector features nearly 
complete solid-angle coverage and can precisely measure electrons, photons, muons, jets and 
missing energy over a large range of particle energies. These broad capabilities of the CMS 
detector will allow the exploration of electroweak symmetry breaking and enable the potential 
discovery of physics beyond the Standard Model 0. 

The CMS trigger and data acquisition systems are responsible for ensuring that data samples 
with potentially interesting events are recorded with high efficiency and good quality. The ex- 
periment has a two-level trigger system, unlike most other hadron collider experiments which 
have more traditional three-level systems. The first level is hardware-based and is called the 
"Level-1 Trigger" (LI) while the second level is software-based and is called the "High-Level 
Trigger" (HLT). 

In late 2008 the CMS Collaboration conducted a month-long data taking exercise known as 
the Cosmic Run At Four Tesla (CRAFT) with the goal of commissioning the experiment for 
extended operation [4J. With all installed detector systems participating, CMS recorded 270 
million cosmic ray triggered events with the solenoid at its nominal axial field strength of 3.8 T. 
In addition to commissioning the experiment operationally, the data collected during CRAFT 
proved invaluable for understanding the performance of the CMS experiment as a whole. 

This paper reviews the implementation of the CMS HLT and its commissioning using CRAFT 
data. For the first time, during CRAFT, the CMS HLT was run over an extended period of time 
utilizing data from most sub-detectors. In addition to the gain in operational experience, the 
data collected proved to be extremely useful for developing monitoring strategies and proce- 
dures. In particular, since the physics data were primarily comprised of muons, muon-based 
HLT algorithms benefited most. 

The paper is organized as follows. The CMS Trigger system is described in Section [2] and an 
overview of the triggers deployed online during cosmic ray data taking is provided in Sec- 
tion |3j The output of the HLT is reviewed in Section [4] while the strategies implemented and 
commissioned for monitoring the performance of the HLT are discussed in detail in Section |5j 

2 The CMS Detector and its Trigger System 

The central feature of the Compact Muon Solenoid (CMS) apparatus is a superconducting 
solenoid, of 6 m internal diameter, providing a field of 3.8 T. Within the field volume are the sili- 
con pixel and strip tracker, the crystal electromagnetic calorimeter (ECAL) and the brass/ scintillator 
hadron calorimeter (HCAL). Muons are measured in the pseudorapidity window \rj\ < 2.4, 
with detection planes made of three technologies: Drift Tubes (DT), Cathode Strip Chambers 
(CSC), and Resistive Plate Chambers (RPC). In addition to the barrel and endcap detectors, 
CMS has extensive forward calorimetry (HF). 

A schematic view of the CMS trigger system is presented in Fig.[TJ The details of the trigger de- 
sign and its implementation can be found in Refs. J5HZI- LI is designed to process information 
from the CMS detector at a bunch-crossing rate of 40 MHz. This rate, coupled with the latency 
of 3 jis, limits the data accessible for processing to the information from the calorimeter and 
muon detectors. Additionally, in order to allow fast processing, these data have coarser gran- 
ularity and lower resolution than the full information from the front-end electronics. The LI 
trigger is built mostly of custom hardware and is designed to reduce the event rate to 100 kHz 



2 



2 The CMS Detector and its Trigger System 



at the LHC design luminosity C = 10 34 cm 2 s 1 . 
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Figure 1: Schematic view of the CMS trigger system. 



The CMS HLT hardware consists of a single processor farm called the "Event Filter Farm". 
The farm comprises PCs mounting 2.66 GHz dual quad-core Intel processors and a total of 16 
GB RAM. The HLT algorithms that run on the farm are required to reduce the event rate to 
0(100) Hz. In traditional three-level trigger systems this rate reduction of 0(1000) is achieved 
in two stages: a customized hardware or software-based Level-2 trigger that quickly provides 
the large rejection factor and a Level-3 processor farm that makes decisions based on more 
sophisticated algorithms. The CMS HLT, in contrast, combines the rejection power and speed 
of a Level-2 trigger with the flexibility and sophistication of Level-3 in a single processor farm. 
The design of the filter farm is described in detail in Ref. [8|. 

Traditional three-level systems have to use "regional" information from specific parts of the de- 
tector to achieve the 0(1000) rejection. Instead, in the CMS two-level design, full-granularity 
data from the whole detector (including the tracker) are available to a standard commercial 
computer. Regional data-unpacking is implemented only for some of the sub-detectors to opti- 
mize the CPU performance of the HLT. The CMS HLT decision can then be based on full reso- 
lution data, and the filtering algorithms have the flexibility to use sophisticated offline-quality 
reconstruction algorithms. Due to the high input rate of 100 kHz which the filter farm needs to 
sustain, this implementation requires significant CPU resources. Therefore, the reconstruction 
algorithms that run during the HLT processing must be optimized for performance in order to 
minimize dead-time and, at the same time, collect all interesting events for physics analysis. 

The data acquisition (DAQ) system commissioned during CRAFT consisted of a total of 275 
builder units that assembled the data fragments into complete events, and 825 filter units for 
the HLT processing. Towards the end of CRAFT, a DAQ configuration of 720 8-core PCs and 
nearly 4500 filter units was successfully tested. This DAQ configuration is planned to be used at 
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LHC startup. With such a configuration, the CPU power available in the filter farm is expected 
to be suitable for handling HLT input rates as high as 50 kHz for an average processing time 
of up to ?»100 ms /event. This estimate includes a conservative factor of two in the average 
HLT processing time which has been measured to be ~43 ms using fully simulated events [7|. 
For LHC operation at high luminosity, the computing power of the filter farm is expected to 
double in size, with the use of more recent CPUs, thereby meeting the design HLT input rate 
of ~ 100 kHz. 

3 HLT during CRAFT 

Candidate HLT "menus" for startup were commissioned during CRAFT and their performance 
was studied. A menu is defined as a set of triggers, each of which consists of several reconstruc- 
tion and filtering modules. An event is stored to permanent storage if it is selected by at least 
one High-Level trigger in the deployed menus. 

The HLT menus deployed during CRAFT were designed to reliably deliver data from cosmic 
rays, regardless of detector conditions. Consequently, these menus were primarily composed 
of "LI pass-through" triggers, which transfer to storage the events accepted by one or more LI 
triggers without additional processing or selection. Triggers with additional selection beyond 
the LI decision were added to the HLT menus throughout the commissioning period. In sub- 
sequent studies of the recorded data, these triggers were useful for physics selection, detector 
diagnostics, and the alignment and calibration of the CMS subsystems. 

Prior to CRAFT, HLT algorithms had been studied using Monte Carlo simulated data. The 
introduction of these algorithms to the HLT menus during CRAFT provided an opportunity to 
test their behavior in the detector environment and improve the selection criteria. 

3.1 CRAFT triggers 

Every event processed by the HLT must first satisfy one or more of the LI trigger requirements. 
The LI Global Trigger (GT) [9| can provide up to 128 trigger algorithms that select an event, 
based on logical combinations of LI trigger objects (such as muons, jets, or calorimeter trans- 
verse energy) with applied selection criteria (energy /momentum thresholds, etc.). Up to 64 
simple on/ off signals from the sub-detectors, or "Technical triggers", can also be added to the 
GT decision. The technical triggers provide additional information that may be used for special 
purposes such as detector diagnostics and monitoring. During CRAFT, only the most inclusive 
LI single-object trigger algorithms contributed to the global LI decision: a muon (with no ad- 
ditional selection), a jet with transverse energy above 10 GeV, or an electron or photon with 
at least 1 GeV of transverse energy. The results of the remaining LI trigger algorithms were 
computed but were not considered in the GT decision, i.e., masked. The full LI results (includ- 
ing these masked triggers) were available to the HLT, thus allowing the more complex HLT 
triggers to be seeded by more restrictive LI trigger algorithms. 

The HLT menus deployed during CRAFT included triggers that can be grouped into two main 
categories: LI pass-through triggers, and triggers that required some additional processing 
beyond the LI decision. All HLT triggers were run over all events accepted at LI. 

For most LI pass-through triggers, the HLT only checked the event type as marked by the GT. 
The HLT accepted events if any of the following LI conditions was satisfied: 

• The event was marked as a "physics" event by the GT. This required a positive re- 
sponse from any of the trigger algorithms or technical triggers that contributed to the 
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GT decision. This ensured that all events accepted at LI were kept for later study. 
These events occurred at a rate of ~ 500 Hz. 

• The event was marked as a "calibration" event. These specially triggered events 
occur at the end of about 1% of all LHC orbits during the 3 ]is gap that is devoid 
of protons in both LHC beams [10 1 . This orbit gap was emulated by CMS during 
CRAFT. All LI "physics" triggers (algorithms or technical triggers) are suppressed 
during a calibration event. These events occur at a rate of 100 Hz and are used for 
detector calibration and diagnostics. 

• The event was marked by the GT as a "random" trigger event. These events are sent 
to the HLT in order to verify the LI decision. The random trigger rate was limited to 
a few Hz by the HLT. 

Three HLT pass-through triggers were implemented, one for each of the above conditions. The 
other HLT triggers included in the CRAFT menus selected sub-samples of the data already 
accepted by these triggers. This allowed the data to be organized into more manageable blocks 
(see Section |4). 

In addition to the LI pass-through triggers described above, several more complicated triggers 
were included in the HLT menus to help with detector diagnostics: 

• A trigger, seeded by random LI triggers, accepted events based on the size of the 
pixel tracker data. This sub-detector has a very low noise rate, such that charged 
particles passing through the tracker noticeably increase the pixel data volume. This 
trigger could be used to validate the LI decision, since such an event, with particles 
crossing the pixel tracker, could be accepted by the HLT even if it was not flagged as 
"physics" by the GT. 

• Similarly, a trigger was designed to accept events with energy deposition in HF ex- 
ceeding the noise threshold. 

• During dedicated runs designed to monitor HCAL noise behavior, the LI trigger 
system was configured to accept events consistent with noise from HCAL photodi- 
odes. HLT CRAFT menus included a pass-through trigger to accept these events. 

In order to commission the HLT reconstruction algorithms in preparation for collisions, physics 
triggers that applied additional selection criteria, beyond the LI seed requirement, were also 
included in the HLT menu. These triggers include selection algorithms that will be used to 
filter collision data, but with relaxed criteria (energy/ momentum thresholds, etc.) in order to 
collect cosmic ray and detector noise data. 

• For events accepted by the GT with a LI jet algorithm, the calorimeter data in the 
vicinity of the LI jet were used to reconstruct jet candidates in the HLT using the 
CMS implementation of the Iterative Cone algorithm IfTTJ . The HLT jet transverse 
energy was required to exceed 12 GeV. 

• Events that triggered the LI jet algorithms were accepted by the HLT without addi- 
tional selection criteria. These events, collected without beam passing through CMS, 
will be used for detector background studies, relevant for example for heavy stable 
charged particle searches in collision data [T2| . 

• For events with an electron/ photon LI seed, the ECAL regions surrounding the LI 
seed were examined and an electromagnetic object was reconstructed using calori- 
meter information only. The event was accepted by the HLT if the electron/ photon 
HLT object had at least 5 GeV of transverse energy. Information from the tracker 
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was ignored. 

• Several muon triggers were included in the CRAFT menus: 

-k The muon detector data were examined and the muon was reconstructed 
from the LI muon seed using this information. Muons with at least 3 GeV/c 
of transverse momentum (pj) were accepted. 

* Starting with the LI seed in the muon detector, the path of the recon- 
structed muon was extrapolated inward to the tracker volume. Informa- 
tion from the reconstructed particle trajectory within the tracker was used 
to improve the estimation of the muon momentum. Events with muons 
with pr > 3 GeV/c were accepted. 

* Alternatively, tracks were reconstructed in the tracker volume without 
seeding from a LI muon object. Events accepted by this trigger, which 
examined any event accepted by LI, provided a source of cosmic rays in- 
tersecting the tracker volume that may have been missed by the standard 
HLT muon selection and can be used to improve the HLT reconstruction 
algorithms. This trigger could only be implemented during commission- 
ing when the HLT input rate is low compared to the pp collision rate. 
In order to process the large number of events expected from LI during 
colliding beam operations, any trigger that implements some non-trivial 
selection criteria in the HLT must first refine the input using LI seed in- 
formation. 

Finally, triggers were included in the HLT menus to satisfy the alignment and calibration needs 
of the CMS experiment. The performance of these triggers will be discussed in Section 5. 

3.2 Online deployment of the HLT menu 

The menu that is deployed online for data taking operations closely resembles the offline-tested 
menu. The HLT uses the same software to run offline on Monte Carlo simulations and pre- 
viously collected events and to record online collision or cosmic ray data. However, a small 
set of technical adjustments is necessary when the menu is moved from the offline computing 
environment to the Event Filter Farm. These adjustments can potentially introduce a mismatch 
between what is tested offline and what is used online. 

The HLT menu is tested on a dedicated section of the Event Filter Farm prior to online deploy- 
ment. This validation farm emulates the behavior of the full Event Filter Farm and is used to 
validate the HLT menu on previously collected data samples. The validation farm comprises 
a Storage Manager (SM) application and an adjustable number of filter units, up to 19, each 
running up to four independent event processing threads. The SM is the last component in 
the data-handling chain and performs two primary functions. The first function is to collect 
events from each Filter Farm HLT process, and store them in files for subsequent transfer and 
processing. The data files are assigned to one or more output streams (see Section 4) according 
to the defined trigger bits and the SM configuration information. The second function of the 
SM is to act as an event server for calibration and monitoring purposes. 

A sample of previously collected data events is examined using the validation farm, the events 
being filtered by the trigger menu under consideration. This procedure validates several as- 
pects of the trigger menu which cannot be otherwise tested offline: 

• the trigger menu behavior, when first deployed on the Event Filter Farm. Due to 
the differences in the online and offline environments, it is possible that an offline- 
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validated menu is incompatible with the online environment; 

• the selection of events by the menu, and whether rates are consistent with expecta- 
tions from offline tests; 

• the correct grouping of data into streams, each containing the specified set of "pri- 
mary datasets" (see Section 4); 

• the communication with the Storage Manager and the production of the appropriate 
output file for each stream. 

The online test also provides a useful opportunity for debugging the candidate HLT menu. The 
Event Filter Farm places problematic events, such as events with corrupt data or reconstruction 
problems resulting in the failure of the HLT filtering process, into a special "Error" stream. The 
content of this stream is analyzed offline in order to diagnose problems and to serve as a test 
for new trigger algorithms. 

During CRAFT operation, a menu was first validated with data on the validation farm and then 



deployed online. The performance of the menu was then monitored as discussed in Section 5.1 



A total of 14 HLT menus were deployed during the CRAFT data taking period. In most cases, 
new menus were created by making small adjustments to the existing menu. These changes 
were prompted by the daily operational goals and could be validated and implemented within 
an hour. By contrast, the HLT menus that included triggers designed to test HLT reconstruc- 
tion algorithms were validated over several days to ensure that they could be deployed online 
without problems. 



4 Streams and Primary Datasets 

As described in Section |3.2} the SM is responsible for collecting events from each Filter Farm 
HLT process and storing them in files for subsequent transfer and processing. Events are 
grouped into a set of streams, with distinct sets of data files produced for each stream. The 
grouping of events is done according to their offline usage. As an example, the primary stream 
is "Stream A", or the "physics" stream, which consists of events which satisfy the needs of 
most offline physics analyses. Other streams serve special purposes such as monitoring the 
performance of the HLT algorithms. 

Different streams must be defined for samples that require special or reduced amount of infor- 
mation ("event content"). This is necessary since the software framework requires both online 
and offline data files to contain homogeneous information. For example, the event content used 
for the physics stream comprises the full detector and trigger raw data, and the LI and HLT 
trigger results. However, streams used for calibration can benefit from storing only a fraction 
of the raw data, while using triggers with higher accept rates than those included in the physics 
stream. This allows the highest possible collection rate for calibration events while respecting 
bandwidth constraints. 

The streams that were defined in the HLT configuration and exercised during CRAFT are listed 
below: 

• Stream A: The primary physics stream, which consists of detector and trigger raw 
data and LI and HLT trigger summary results. The full list of physics triggers feeds 
this stream. 

• DQM: Used to monitor the performance of the detector, the LI trigger and the HLT 
algorithms. All relevant by-products of the HLT reconstruction are stored in order to 
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allow detailed online and offline validation. These reconstruction by-products are 
objects created by the HLT processing, contain detailed information regarding the 
execution of the algorithms, and can double the size of the data stream for certain 
types of events. These by-products are not stored in the Stream A output. 

• Calibration: Used to collect "calibration" type Ll-triggered events during the orbit 
gaps (see Section |3T|. The calibration stream provides events for monitoring and 
calibrating the various sub-detectors. 

• AlCaRaw: A group of streams used to collect data for calibrating the ECAL and 
HCAL detectors. Typically, these streams are fed by high rate triggers which have 
reduced event content. During collisions, these data will be used for providing new 
calibration and alignment constants for use by offline reconstruction algorithms.The 
AlCaRaw streams are described in more detail in Section l4!2l 

During CRAFT, a total of seven streams (Stream A, DQM, Calibration, and 4 AlCaRaw streams) 
were produced. Upon transfer to the CERN "Tier-0" computing site, the data corresponding to 
the various streams was reformatted and split into various "Primary Datasets". These datasets 
are defined, within a stream, by grouping a set of triggers that perform similar selections (e.g., 
all muon-based triggers within Stream A were grouped into the "Cosmics" dataset). The pri- 
mary datasets are intended to provide small-sized samples suitable for specific physics analy- 
ses. 

The definition of multiple streams and primary datasets during CRAFT and their subsequent 
offline processing helped in the commissioning of the full workflow that will be used at LHC 
startup. 

4.1 DQM stream 

To perform data quality monitoring (DQM), a special stream was used during CRAFT. This 
stream contained the same triggers as the primary physics stream, Stream A, with two main 
differences. The DQM stream included events selected by the calibration triggers, which were 
part of the calibration stream. This allowed the DQM team to check and monitor the quality of 
the data used for calibration purposes. In addition, the stream included the HLT reconstruction 
by-products. These by-products are essential for monitoring the proper performance of the 
HLT algorithms and can be effectively used for debugging purposes. 

The DQM stream is required to provide sufficient data for online DQM (see Section [5J] |. The 
rate of the stream is determined mainly by the time taken to reformat the data before it is fed 
to the DQM applications. During CRAFT the rate of the stream was 25 Hz, but for the final 
operation during collision data taking, a rate of about 60 Hz or higher is expected. 

4.2 AlCaRaw streams 

The term "AlCaRaw" is used to designate high-rate triggers that provide events for detector 
alignment and calibration. The name consists of two parts: "AICa" indicates that the events are 
used for alignment and calibration, and "Raw" indicates that the events are selected directly at 
the raw data level. 

Alignment and calibration techniques typically require large size samples. However, only a 
small portion of the event data is used, i.e., the full event is not needed for most alignment 
and calibration purposes. Usually, minimum bias events serve this purpose best. These have 
high rates and hence directly compete with the triggers needed for physics analyses. In or- 
der to collect the statistics required for alignment and calibration, without saturating the data 
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taking bandwidth, dedicated triggers record events and carry out data reduction so as to store 
the minimal amount of information needed for the calibration or alignment processes. Data 
collected by these triggers are used for the calibration of the ECAL and HCAL sub-detectors. 

Four AlCaRaw streams were exercised during CRAFT: 

• ECAL 7T°: uses a constraint on the invariant mass formed from two photons to com- 
pute inter-calibration coefficients for the electromagnetic calorimeter. The selection 
of 7T° candidates is based on two isolated photon candidates where the invariant 
mass of the two photons is close to the tc° mass. The output of the AlCaRaw trigger 
contains only the reconstructed hits forming the n° candidates. The typical event 
size is around 1.5 kB which corresponds to about one percent of the full event size. 

• ECAL (p symmetry: relies on the azimuthal ((p) invariance of the energy deposition 
in physics events. This trigger records all reconstructed hits in the event. The typical 
event size is around 5 kB. 

• HCAL (p symmetry: relies on the (p invariance of the energy deposition in physics 
events. In order to extract low energy deposits, the hadron calorimeter is required 
to operate in non zero-suppressed mode and the output contains mainly HCAL raw 
data. The typical event size is large due to the non zero-suppressed HCAL output 
and is approximately 180 kB. 

• Isolated track: uses isolated tracks to measure calorimeter response to single pions. 
The trigger requires an isolated track with a large transverse momentum (at least 
lOGeV/c). Recent studies have shown that this trigger can be included as an HLT 
trigger since the rate is low (~10 Hz) even though the event size is rather large 
(hundreds of kB). 

For the CRAFT exercise, the AlCaRaw triggers were used with some relaxed criteria in order 
to record events for testing and validating the workflow for calibration. 



5 Monitoring of the HLT performance during CRAFT 

During CRAFT many tools for monitoring the High-Level trigger were developed and tested. 
In this section the monitoring of the performance of the muon High-Level triggers is presented, 
since they were more relevant for the cosmic ray data. 

5.1 Online DQM 

Trigger monitoring at CMS is performed at two different levels: the first makes use of moni- 



toring software run directly in the filter units (see Section 5.1.1 1, the second consists of a quasi- 
online monitoring system which reads events after HLT processing (see Section 5.1.2) . The 
quasi-online monitoring is preferred, since there are no limitations on the allowed bandwidth, 
which is the case for software running in the HLT. 

5.1 .1 DQM in the High-Level Trigger 

In the HLT, the filter units process events at rates up to 100 kHz and produce a limited number 
of histograms. The histograms are delivered from each filter unit to the storage managers at the 
end of each luminosity section (corresponding to roughly 93 seconds of data taking). Identical 
histograms are then summed together and sent to a storage manager proxy server, which saves 
the histograms to files and serves them to DQM consumer applications along with the events. 
The advantage of monitoring the data in the filter units is that the DQM consumers can access 



5.1 Online DQM 



9 



all the events processed by the HLT, including events that will eventually be rejected at this 
stage. The main focus of the trigger DQM modules in the filter units is to monitor LI and HLT 
scalers, and compute rates and rejection factors for each trigger algorithm and filtering stage. 
During CRAFT fewer than ten histograms were created in this way. 

5.1.2 Quasi-online DQM 

In the HLT a dedicated DQM stream can be implemented that selects any type of events, based 
on HLT decisions. During CRAFT the HLT typically did not reject any event and an unbiased 
fraction of events were randomly selected to be sent to the DQM stream. This stream provides 
detector raw data and, on explicit request, additional HLT information. The DQM applications 
receive event data coming from the storage manager proxy server using the HTTP protocol. 
These events can be delivered at a configurable rate, which was set to 25 Hz during CRAFT. 
The quasi-online monitoring of the trigger includes monitoring of the LI hardware as well as 
monitoring of the level of agreement between emulated data and hardware data. The emu- 
lator software reproduces the hardware decision for a given trigger configuration, is used to 
compute trigger efficiencies offline, and allows comparisons with hardware data. 

5.1 .3 Monitoring the performance of the Muon HLT 

The muon HLT algorithms subdivide the HLT processing into two logical steps: Level-2 (L2) 
and Level-3 (L3). This nomenclature is common to all HLT triggers in CMS and echos a tradi- 
tional three-level trigger system. L2 uses LI muons as seeds and incorporates the full resolution 
of the muon system and the calorimeters. L3 includes the information from the tracker. The 
muon trigger reconstruction detects muons by linking tracks in the muon system with deposits 
in the calorimeter consistent with minimum-ionizing particles and with tracks in the inner 
tracking system. 

Muon HLT triggers used during CRAFT were those that were designed for use during LHC 
collisions. Since cosmic ray muons only rarely cross the beamline, most of them will not satisfy 
these muon HLT requirements. Moreover, the muon reconstruction algorithm used in the L3 
step of the HLT only performs the tracking within restricted regions around the L2 muon trajec- 
tory. This restriction shortens the time required for the L3 processing and allows these triggers 
to meet the timing constraints of the HLT. This restricted regional tracking algorithm uses the 
beam spot position to determine the tracking regions, which is not appropriate in the case of 
cosmic ray muons. Even though not many events are expected to satisfy the muon triggers that 
require tracking, those that do satisfy this requirement can be used to collect data for studying 
the performance of the muon HLT algorithms. 

The stability of the various triggers during collision data can be measured in terms of trigger 
rate and as a function of the measured luminosity. For cosmic data taken during CRAFT the 
stability was monitored by comparing the rate of each trigger with that of the most inclusive 
one, HLT LIMuOpen, a trigger requiring a LI muon without any momentum or quality restric- 
tions. The performance of the latter was examined separately. The same scheme will be used 
with collision data and will provide a measurement of the trigger performance which does not 
depend on uncertainties of luminosity measurements. Figure [2] (left) shows the rate for various 
muon triggers, with respect to the HLT_LlMuOpen trigger, as measured in the online DQM 
during a good quality run of 50,000 events. This figure serves as a reference distribution and is 
used to estimate the stability of HLT performance in other runs. The relative trigger rates de- 
pend on trigger and sub-detector configurations; therefore, a few reference distributions, one 
for each possible configuration, were prepared. The HLT trigger efficiency cannot be precisely 
measured in online DQM, but the stability of trigger performance can be monitored by com- 
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paring the number of LI seeds and the number of L2 objects for a selected trigger as shown in 
Fig. [2] (right). The figure represents the fraction of L2 muons with respect to the number of LI 
muons, as a function of the LI muon pseudorapidity rj. Only events selected by a L2 trigger, 
L2Mu3, are considered. This figure serves as a reference distribution and changes to it indicate 
potential modifications in the performance of the muon trigger logic. A more detailed study of 
L2 and L3 muon trigger efficiencies and a comparison with offline reconstruction algorithms 
are presented elsewhere [ 131 . 
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Figure 2: Relative rate of the listed triggers with respect to the HLT_LlMuOpen trigger (left) 
and the fraction of L2 muons to LI muons as a function of the LI muon pseudorapidity for 
events selected by the L2Mu3 trigger (right). 



5.2 Offline low-latency high-statistics data quality monitoring 

The offline reconstruction process at Tier-0 allocates computing resources which can be de- 
ployed for systematic data quality monitoring. "Offline DQM" software applications, similar 
to those described for the online DQM, can be introduced as part of the offline reconstruction 
task itself. The primary disadvantage of monitoring data in this fashion is the latency of offline 
reconstruction relative to raw data collection, which can be as long as a few days. The comple- 
mentary advantages of offline DQM are twofold: all reconstructed events can be monitored, so 
that less obvious, slowly developing, or rare problems can be identified; and the differences be- 
tween online and offline physics object reconstruction can be readily studied. During CRAFT, 
offline DQM measuring of the HLT performance was exercised as part of all offline reconstruc- 
tion operations, and results were available for study as part of the central DQM infrastructure 
for CMS. 

5.2.1 Software 

The software applications for offline DQM are identical in structure to those for online DQM 
discussed previously. The offline DQM produced the following histograms: LI trigger hard- 
ware performance plots, comparison of emulated LI data with recorded LI data, as described 
in the previous section, and plots of basic properties of all objects (muons, jets, etc., from LI, 
HLT, and offline reconstruction) passing each of the HLT paths. Plotted object properties in- 
clude transverse energy spectra, rj-(p occupancy, and object multiplicity. The offline DQM soft- 
ware parses the HLT menu at run time and automatically configures the correct set of plots 
for each trigger path in the menu. Figure [3] shows the occupancy of muons which passed a LI 
muon path for a single CRAFT run. With the large sample size available offline, it is possible 
to identify regional variations in muon trigger occupancy from run to run. 



5.2 Offline low-latency high-statistics data quality monitoring 
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In addition to object properties recorded for each path, the offline DQM software can require 
the objects to also pass a (configurable) reference path, allowing for object sampling which is 
unbiased for trigger efficiency studies. For example, a minimum-bias reference trigger path 
could be required, and offline muon candidates can be plotted for all events in this reference 
path, and similarly for all events which pass some muon trigger test path. The quotient of 
the transverse energy spectra of these two distributions is an estimator of the test path trigger 
efficiency as a function of transverse energy for muons reconstructed offline. In preparation 
for proton collisions, more specialized plots will be introduced. Each trigger object type will 
be analyzed separately, selecting high-purity and low-purity samples for signal efficiency and 
background rejection studies. 



CMS 2008, Run 66676 




Figure 3: Example offline DQM occupancy plot for LI muon candidates from a single CRAFT 
run, for events with high-quality cosmic muons reconstructed offline. 

5.2.2 Usage for CRAFT 

The offline DQM applications ran regularly as part of offline reconstruction throughout the 
CRAFT run, and the results were available for viewing by offline DQM shifters. Automated 
quality tests and data certification were not yet deployed but are planned for data taking. The 
function of the offline DQM is largely to confirm with higher precision the online DQM results. 

Latency for offline DQM studies ranged from as little as an hour to as long as a few days, 
depending on the reliability of the offline reconstruction at any point in time. No problems 
were originally diagnosed offline, but it is anticipated that the increased HLT menu complexity 
and longer integration times of proton collision runs will require offline monitoring to discover 
more rare or more slowly developing problems. 

Other studies of the trigger conducted for the CRAFT run included: emulation of the HLT algo- 
rithms offline and comparing the results with the actual HLT data; studies of LI and HLT per- 
formance of cosmic muons; studies of the effect of calorimeter noise on LI calorimeter triggers, 
and observation of effects of noise and /or cosmic rays when a menu anticipated for proton 
collisions was deployed. 



12 



5 Monitoring of the HLT performance during CRAFT 



5.3 Monitoring of the CPU performance 

The average CPU time taken to accept or reject events at the HLT depends significantly on 
the type of event accepted by the LI trigger. This in turn depends on the sub-detectors in- 
cluded during the global data taking process, the LI and HLT prescales, and other operating 
conditions. In order to track the behavior of the HLT CPU time under distinct configurations, 
incremental CPU performance studies were carried out using a few runs with different menus. 
The results for two of these runs are presented in this section. 

Runs 66279 and 68021 were recorded during CRAFT using two different HLT menus with dif- 
ferent numbers of HLT paths. The sub-detector configuration for these runs is summarized in 
Table [I] In each case, the performance of the full HLT menu was determined by measuring 
the total HLT processing CPU time when running on a subset of the Ll-accepted events. This 
total CPU time (measured on a Dual-Core 5160 Xeon processor running at 2.0 GHz) includes 
both the data-unpacking time and the time taken to execute the algorithms.The results scale 
by 0.7 since the machines used in the Filter Farm run on Quad-Core E5430 Xeon processors at 
2.66 GHz. 

Table 1: Run configuration and total CPU time for runs 66279 and 68021 taken during CRAFT. 





Run 66279 


Run 68021 


Events used 


30000 


30000 


Sub-detectors 


No ECAL, no tracker 


No HF 


included 


no RPC 




in run 






Magnetic field 


0T 


3.8 T 


# of trigger paths 


16 


24 


Average CPU time 


15 ms 


47 ms 



The ECAL, the tracker, and the RPC sub-detectors were not included in Run 66279. Further- 
more, the HLT menu used was rather simple, as it contained only 16 triggers, which allowed 
primarily low quality muon objects to pass. 

Run 68021 incorporated most of the detectors that were missing in the previous run; how- 
ever, the forward calorimeter (HF) was not included. In addition, an increase in the number of 
triggers from 16 to 24 added complexity to the HLT menu. The new triggers included muon 
triggers with low thresholds that used tracking information, a jet trigger, and a low threshold 
trigger based on electromagnetic energy. The complexity of the new algorithms (those using 
tracking information in particular) and the inclusion of the missing sub-detectors are reflected 
in the three-fold increase of the average CPU time. Taking runs with noisy sub-detectors al- 
lowed gauging of the impact of non-optimal operating conditions on the performance of the 
HLT. After a few such runs, only one algorithm was found to be non-optimal in terms of CPU 
performance. The data collected during these special runs were used to develop more opti- 
mized algorithms that were subsequently deployed online. In one other case, corrupt data 
from one of the sub-detectors resulted in unexpectedly verbose error logging which in turn 
had an adverse effect on the the HLT processing time. Since this algorithm had previously 
been extensively and successfully tested on simulated events, this incident reiterated the im- 
portance of commissioning the HLT algorithms with data. Following the CRAFT data taking 
period all HLT algorithms were scrutinized in detail to ensure that error logging was kept to a 
minimal and that corrupt data were instead detected via DQM applications. 

The unpacking of LI data took 3 to 4 ms and the remaining CPU time was taken up by sub- 
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detector unpacking (based on LI decisions) and the execution of the HLT algorithms. The LI 
unpacking time was approximately the same for all the studies described in this section. This is 
the expected behavior since the addition of new triggers to a menu does not increase the CPU 
time for the LI data unpacking, as this is carried out only once per event. 

This monitoring of the CPU performance of the HLT allowed testing of the incremental stability 
of the system under pronounced variations of conditions and sub-detector configurations. 

6 Summary and Conclusions 

The commissioning efforts for the High-Level Trigger in 2009 have built on the progress made 
during the 2008 CRAFT period. During CRAFT, physics-based selection criteria beyond the LI 
decision were integrated into the HLT menus and the latter were run successfully during an 
extended period of time. The ability to reliably and efficiently collect physics-enriched datasets 
is critical for collisions, and CRAFT proved invaluable for commissioning HLT operations. 
We monitored the overall performance of the HLT and confirmed its dependence on trigger 
design: triggers that are not robust or are not optimized for online performance greatly affect 
the stability of HLT operations. 

These lessons have significantly influenced recent commissioning work. The HLT menus for 
collisions have been redesigned in preparation for data taking with greater emphasis being 
paid to simple, inclusive menus with a small set of robust triggers. The triggers present in 
these new, lean menus are being scrutinized to maximize trigger performance; this includes 
optimizing the choice of LI seeds for each trigger, so as to select those events considered most 
likely to pass the HLT selection algorithms. Reconstruction algorithms are also being analyzed 
for performance to ensure that fast, reliable trigger decisions can be reached regardless of de- 
tector conditions. 

Although individual triggers have been extensively tested on simulated events, changes in data 
quality delivered by the CMS detector (e.g., noisy or corrupt data) can dramatically influence 
the trigger behavior. For this reason, HLT DQM operations serve as an important check of the 
trigger performance. Any change in the trigger behavior can be quickly detected and prompt 
experts to implement fixes to the existing triggers or design more robust algorithms for fu- 
ture menus. HLT DQM has improved considerably since 2008 as a direct result of the CRAFT 
experience. 

The alignment and calibration workflows were successfully validated during CRAFT. The Al- 
CaRaw datasets were rapidly made available and were used to study detector performance. 
This demonstrated that the HLT can efficiently provide large datasets that can be used for a 
rapid calibration of the CMS detector during collision data taking. These efforts have contin- 
ued in 2009 and remain a successful part of the HLT commissioning process. 
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